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1.0 Introduction 

This section summarizes the verification and validation (V&V) of recently implemented, NASA- 
supported enhancements to the decision support tools of the Production Estimates and Crop Assessment 
Division (PECAD). As previously discussed, the implemented enhancements include operationally 
tailored Moderate Resolution Imaging Spectroradiometer (MODIS) products and products of the Global 
Reservoir and Lake Monitor (GRLM). The MODIS products are currently made available through two 
separate decision support tools: the MODIS Image Gallery and the U.S. Department of Agriculture 
(USDA) Foreign Agricultural Service (FAS) MODIS Normalized Difference Vegetation Index (NDVI) 
Database. Both the Global Reservoir and Lake Monitor and MODIS Image Gallery provide near-real-time 
products through PEC AD’ s CropExplorer. This discussion will address two areas: 

1 . Assessments of the standard NASA products on which these enhancements are based. 

2. Characterizations of the performance of the new operational products. 

2.0 V&V of Operationally Tailored MODIS Products 

As mentioned above, two operationally tailored products are being provided to enhance PECAD ’s overall 
set of decision support tools: the CropExplorer MODIS Image Gallery and the USDA FAS MODIS 
NDVI Database. The CropExplorer MODIS Image Gallery is accessed through the CropExplorer 
interface and provides near-real-time access to processed MODIS image subsets. The CropExplorer 
MODIS Image Gallery is actually li nk ed to a selected set of the imagery available through the Goddard 
Space Flight Center (GSFC) MODIS Rapid Response System. PECAD’s decision support requirements 
related to near-real-time MODIS images are detailed in Table 1. 


Table 1 . PECAD accuracy & delivery requirements for near-real-time 
imaging (NASA, 2004a). 


Category 

Requirement 

Product accuracy 

Undetermined 

Data time step 

1 day 

Latency 

6 hours or less 

Coverage 

All land regions important for crop 
production and food security 


* To be evaluated in FY 2006. 


In addition to quasi-global, near-real-time imaging, PECAD also requires a quasi-global NDVI image and 
time series database. Multi-day compositing is necessary for this product to reduce the impact of clouds, 
bidirectional reflectance distribution function effects, and other confounding factors. The requirements for 
an NDVI database as captured in the earlier decision support system evaluation are summarized in Table 
2 . 
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Table 2. PECAD accuracy and delivery requirements for an NDVI 
database (NASA, 2004a). 


Category 

Requirement 

Product accuracy 

Must be consistent with existing 
NDVI database going back to 1981 

Data time step 

1 0 days* 

Latency 

72 hours or less (measured from the 
end of the compositing period) 

Coverage 

All land regions important for crop 
production and food security 


* Considerable discussion has been held about this requirement since the release of the 
decision support system evaluation. At present this is not a firm requirement. Compositing 
intervals of 8, 10, and 16 days are under consideration. 


Neither Table 1 nor Table 2 addresses spatial or geopositional requirements, which has been a matter of 
on-going discussion between PECAD and NASA. PECAD uses remote sensing data with ground sample 
distances (GSDs) ranging from less than a meter to multiple kilometers. For continuous regional 
monitoring, the MODIS GSD range (250-1000 m) has been judged to be sufficient, recognizing the 
necessary trades to achieve temporal requirements. However, field level analyses are not possible at this 
GSD range, so an idealized GSD would be tens of meters or less. Geopositionally, the PECAD need to 
perform temporal analysis and to cross-reference among several datasets implies a geopositional accuracy 
of one-half GSD or better. 

2.1 Calibration and Validation of MODIS Sensor and Standard Products 

The on-orbit calibration and validation of the MODIS sensor and standard MODIS products was dealt 
with comprehensively (for Terra MODIS) in a special issue of Remote Sensing of Environment in 2002. 
Several intrinsic properties of interest to operational users were characterized. Wolfe et al. (2002) 
reported a geolocational accuracy of 50 m (1 a) at nadir. For MODIS Bands 1 and 2 (the red and near- 
infrared bands used for PECAD tailored products), Rojas et al. (2002) found that the spatial response was 
consistent with prelaunch performance in the cross-track direction but was a bit degraded in the along- 
track direction. Huete et al. (2002) demonstrated that MODIS NDVI performed at higher fidelity than 
Advanced Very High Resolution Radiometer (AVHRR) NDVI. In a discussion of atmospheric correction 
for standard MODIS products, Vermote et al. (2002) showed a clear example of how not correcting for 
aerosol can dramatically affect the distribution of NDVI values throughout a scene. 
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Further work has been done regarding the 
cross validation of MODIS NDVI with other 
sensors. Gallo et al. (2004) found a linear 
relationship between AVHRR and MODIS 
16-day composite time series with overall 
coefficient of variation (r 2 ) above 0.9 
(considering 9 diverse land cover types). 

Working directly in support of the 
NASA/FAS partnership the Global Inventory 
Monitoring and Modeling Studies (GIMMS) 
group at NASA/GSFC has found similar 
results studying AVHRR, MODIS (MOD 1 3 
standard products), and SPOT 
VEGETATION (Justice, 2005). In Figure 1, 

NDVI time series are displayed from 
GIMMS estimates for one crop production 
region in South Africa and for another region 
in eastern Australia (the green line in each 
plot is MODIS Enhanced Vegetation Index, 
which is not used in current MODIS products 
tailored for PECAD). All of the NDVI time 
series have correlation coefficients above 0.9 
relative to each other. 

2.2 Characterization of the MODIS 
Operationally Tailored Products 

The NDVI values in the USDA FAS MODIS 
NDVI Database are the same as the standard 
MODIS product NDVI values; the values are 
simply masked or summarized according to 
PECAD needs. Consequently, the standard 
MODIS NDVI validation work applies. 

Characterization of CropExplorer MODIS 
Image Gallery product accuracy is of 
questionable value because the final products 
are really pictures with visual as opposed to 
quantitative information content. As a result, 
the remainder of this section focuses on 
assessment of product performance in terms 
of delivery and coverage. 

2.2.1 MODIS Product Delivery 

The MODIS Rapid Response System product 
delivery was observed for 15 delivery days starting with June 23, 2005, (Julian day 173) through July 6, 
2005 (Julian day 187). On July 2, 2005, an interruption of data lasted for more than 2 days. This problem 
affected all the data that was passed through the MODIS Rapid Response System to the CropExplorer 
MODIS Image Gallery and was reported on the MODIS Rapid Response System Status page 
(http://rapidfire.sci.gsfc.nasa.gov/status/) as shown in Figure 2. 


Maize Triangle ZA (mixed) VI 



Victoria All (wheat) VI 



Figure 1 . Vegetation index time series for the Maize 
Triangle region of South Africa and for the Victoria 
wheat-growing region of eastern Australia. 
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Date 

Description 

2 July 2005 
2005/183 

Some current data is not available due to hardware issues. 


Figure 2. July 2005 data interruption as reported on the MODIS Rapid Response System Status page. 


Even five days after the July 2, 2005, interruption, the MODIS Rapid Response Delivery System delivery 
times were notably out of step with pre-interruption system performance, so only the first 1 0 days of 
observation were used to estimate “typical” system performance. 


The estimated system latency for 59 tiles with an Aqua and a Terra delivery opportunity for each of 10 
days (1180 distinct delivery opportunities) was 4.7 hours. The distribution of lag times was heavy tailed 
on the high side (Figure 3) with just over 65 percent of the MODIS Rapid Response System information 
available within the 6-hour requirement stated by PECAD and 80 percent within 24 hours. Even within 
the observed 1 0-day window, one 
significant interruption of delivery 
began on June 26 and ended on 
June 27. This event was not 
reported on the MODIS Rapid 
Response System Status page. 

A systematic relationship existed 
between lag time and the time of 
day in which a particular subset- 
tile was acquired and processed. 

Figure 4 shows that the lag time 
trend reaches a minimum of about 
3 hours near 4:00 AM - 6:00 AM 
(UTC) and then rises slowly 
through the course of a day, 
reaching a maximum of more than 
5 hours between 6:00 PM and 
midnight (UTC). The geographic 
distribution of delivery and lag 

times for Terra is shown in Figure 5. The geographic distribution of delivery and lag times for Aqua is 
shown in Figure 6. 



Figure 3. Distribution of CropExplorer MODIS Image Gallery 
delivery time lag from 23 June 2005 through 6 July 2005. 
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Figure 4. Variation of CropExplorer MODIS Image Gallery delivery time lag by time of day. 


Taken as a whole, the CropExplorer MODIS Image Gallery latency is meeting the stated PECAD 
requirement of 6 hours. Nominally, the data time step is being met, but lapses in availability (both 
identified and unidentified in the MODIS Rapid Response System Status page) are a cause for concern. 

The USDA FAS MODIS NDVI Database was not monitored in as careful a manner because it is more 
developmental in nature. However, it was observed that through June and July 2005, the shortest time 
lapse between end of compositing window and USDA FAS MODIS NDVI Database timestamp was 
about 9.5 days. This lapse has not yet met the project goal of 7 days, nor the PECAD goal of 72 hours 
stated in the DSS evaluation (NASA, 2004a). Given the 16-day time step and a latency of about 10 days 
(or more), the chance of PECAD analysts having more than one NDVI Database reading for the month 
leading to lock-up is small. 
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Figure 5. CropExplorer MODIS Image Gallery delivery and lag times for Terra. 



Figure 6. CropExplorer MODIS Image Gallery delivery and lag times for Aqua. 
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2.2.2 MODIS Product Coverage 

The global footprints of the CropExplorer MODIS Image Gallery and the USDA FAS MODIS NDVI 
Database are shown in Figure 7. The USDA FAS MODIS NDVI Database essentially captures all of the 
economically important global crop production regions, but for some reason the Database omits a few 
regions of interest for food security including the Congo basin, the Horn of Africa, and Madagascar. The 
CropExplorer MODIS Image Gallery misses a few crop production areas of some economic importance 
including southeast Asia, parts of central Asia, Central America, parts of northern South America, and the 
northernmost productive region of the Great Plains of North America. Note that several subset-tiles have 
been added in northern Africa. These subset-tiles were not present at the beginning of the delivery time 
observation window, so they are not part of that analysis. Additionally, at the MODIS Rapid Response 
System site, subset-tiles are identified for FAS associated with Myanmar and southeast Asia that are not 
being passed through to the CropExplorer MODIS Image Gallery. 



Figure 7. Comparison of CropExplorer MODIS Image Gallery and MODIS NDVI Database 
geographical footprints as of July 2005. 


Overall, the MODIS operationally tailored products have nearly met the PECAD coverage requirements 
for economically important crop production, but important coverage gaps exist in terms of PECAD’s food 
security mandate: most notably Central America and northern South America for the CropExplorer 
MODIS Image Gallery and the Congo basin for all MODIS products delivered to PECAD. 
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3.0 V&V of Global Reservoir and Lake Monitor Products 

PECAD ’s decision support requirements related to surface water level variation are detailed in Table 3. 


Table 3. PECAD accuracy and delivery requirements for surface water level 
variation (NASA, 2004a). 


Category 

Requirement 

Surface water level relative accuracy 

10 cm 

Data time step 

1 0 days 

Latency 

7-14 days 

Coverage 

Surface waters in all land regions important 
for crop production and food security 


3.1 Calibration and Validation of Jason-1 Radar Altimeter 

Global Reservoir and Lake Monitor products are based on the measurements of the Jason- 1 and 
TOPEX/Poseidon satellite radar altimeters (USDA, 2004). These are joint missions between NASA and 
the French space agency, the Centre National d’Etudes Spatials (CNES). The primary purpose of these 
altimeters is to increase understanding of ocean surface topography. Their principal measurement is sea 
surface height (Menard et al., 2003). The ranging approach for determining sea surface height differs in 
some particulars from the approach for determining surface water levels, but the two measurements are 
based on many common principles. Given that Jason- 1 and TOPEX/Poseidon were designed for open sea 
measurements, the accuracy achieved by these instruments in measuring sea surface height sets the 
optimistic bound on accuracy that can be expected in measuring surface water levels. 

After the launch of Jason- 1 on December 7, 2001, about 10 months were dedicated to calibration of the 
new instrument and assessment of its performance. For much of that period, the Jason- 1 and 
TOPEX/Poseidon satellites were operated in a common orbit with about one minute of separation 
between the two platforms, allowing an updated assessment of the TOPEX/Poseidon performance as well 
as a cross-validation between the two systems. 

Three primary levels of distributed products were evaluated: 

1. Geophysical Data Record (GDR) - final product based on ground processing, available in 30 days; 

2. Interim Geophysical Data Record (IGDR) - intermediate products based on a preliminary orbit 
determination, available in 3 days; and 

3. Operational Sensor Data Record (OSDR) - near-real-time products based on onboard processing, 
available in 3-5 hours. 

Of these products, the GDRs and IGDRs are relevant to the Global Reservoir and Lake Monitor. The 
GRLM used GDRs for TOPEX/Poseidon data to establish historical data going back to 1992 and used 
IGDRs for Jason- 1 data to provide a near-real-time product with a latency on the order of 10 days. 

Menard et al. (2003) summarized the Root-Sum-of-Squares (RSS) estimates of Jason- 1 sea surface height 
errors as 3.3 cm for the GDR product and as 3.9 cm for the IGDR products. Working in diverse locations 
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including the Harvest platform near California (Haines et al., 2003), Corsica (Bonnefond et al., 2003), and 
the Bass Strait on the north side of Tasmania (Watson et al., 2003), the individual calibration/validation 
teams commissioned by CNES and NASA all achieved sea-based results consistent with the overall 
estimates. Each of these studies also measured TOPEX/Poseidon errors and found them to be in the same 
3-4 cm range. Chambers et al. (2003) cross validated Jason- 1 against TOPEX/Poseidon. Their estimates 
of individual sensor error were similar and they found the magnitude of the global bias between the 
sensors to be 14-15 cm. Residuals showed dependency on significant wave height, with local biases near 
coastlines and other more placid waters being somewhat lower and local biases in rougher waters being 
somewhat higher. This dependency on significant wave height accounts for some differences in bias 
estimates at the fixed sites, and it has implications for surface water level estimates since surface waters 
frequently have smaller wave heights than open ocean. 

Further highlighting the challenge of altimetry over surface waters, one Jason- 1 calibration/validation 
effort studied sea surface height as measured over the Great Lakes with a special focus on a specific 
calibration site on the Ohio shore of Lake Erie (Shum et al., 2003). After accounting for instrument bias, 
the Jason- 1 residuals with Great Lake tidal gauges ran between 3 and 5 cm, which is consistent with other 
results. However, the relative bias between Jason- 1 and TOPEX/Poseidon was estimated to be about 8 
cm, which is 6 cm less than the global bias. Only a small portion of the difference could be attributed to 
wave height effects. Ultimately, the study could not explain the difference. 

3.2 Characterization of the Global Reservoir and Lake Monitor 

An initial verification and validation effort for the GRLM decision support tool was carried out in 2004 
(NASA, 2004b). Though somewhat early in the implementation phase for the tool, this effort touched on 
all the requirements mentioned in Table 3. Since the initial verification and validation effort was 
concluded, the tool has matured in terms of delivery and coverage, so requirements related to these 
properties were re-evaluated through the month of May 2005. As for the product accuracy requirement, 
the fundamental algorithm is essentially unchanged, so the results from 2004 will simply be summarized 
here. For further detail, refer to the online version of NASA 2004b. 

3.2.1 GRLM Accuracy 

The GRLM root mean square accuracy estimates over several North American lakes are summarized in 
Table 4. Essentially, the Great Lakes can be considered representative of the larger water bodies in the 
GRLM. Great Lakes errors ran between 5 and 7 cm. Lake Winnebago and Lake of the Woods may be 
somewhat representative of smaller water bodies. Each estimate represents only a handful of potential 
readings per pass. Lake Winnebago error was calculated to be 27 cm and Lake of the Woods error was 
computed to be 26 cm. Lake Powell represents the most challenging type of lake or reservoir. The lake 
has very few potential readings per pass and it is surrounded by mountainous terrain. Lake Powell error 
over the period assessed was about 140 cm. 

Note the Great Lakes errors computed for the GRLM are somewhat larger than those reported by Shum et 
al. (2003). This discrepancy may be explained by the earlier study’s focus on the verification of sensor 
performance and by corrections for the sea surface height estimates being carefully tailored for the Great 
Lakes. The 2004 GRLM verification effort made no special corrections to the data because the focus of 
that study was the GRLM operational product accuracy. 
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Table 4. Verified GRLM performance over select North American surface water bodies. 


Water Body 

Surface Area 
(www.worldlake.org) 
(km 2 ) 

o 

(outliers 

removed) 

(cm) 

Passes Used in 
Assessment 

Outlying 

Passes 

Removed 

Lake Superior 

82,100 

7.3 

81 

0 

Lake Michigan 

57,800 

5.3 

75 

0 

Lake Huron 

59,600 

6.2 

80 

2 

Lake Erie 

25,700 

6.5 

73 

1 

Lake Ontario 

18,960 

4.5 

79 

3 

Lake Winnebago 

557 

27.0 

24 

0 

Lake of the Woods 

1,900 

26.1 

42 

1 

Lake Powell 

500 

140.6 

244 

6 


NOTE: All lakes and reservoirs were characterized using Jason-1 IGDR-based estimates with the exception of Lake 
Powell. No Jason-1 -based estimates existed for Lake Powell as of May 2004, so TOPEX/Poseidon GDR-based estimates 
were used. 


While not all lakes meet the stated FAS accuracy requirement, perhaps the accuracy requirement should 
be revisited. For deep, narrow reservoirs, fairly large height uncertainty may be acceptable because the 
information regarding water volume is still readily discernable. For shallow, closed lakes (e.g., Lake 
Chad), the current standards are appropriate. Furthermore, FAS lake water height requirements should be 
developed in more detail to include a comprehensive listing of water bodies that FAS believes should be 
monitored. 

3.2.2 GRLM Delivery 

The latency of the GRLM products decreased from 10.3 days to 5.8 days between May 2004 and May 
2005. Latency was defined as the mean lag time between data acquisition and GRLM product upload. The 
maximum lag time in May 2004 was 19.2 days, but in May 2005 the maximum lag time was reduced to 
10.4 days. This dramatic improvement in delivery time was accomplished by reducing the average time 
between GRLM updates from 16 days to 6 days. This improvement clearly meets the stated FAS 10-day 
latency requirement for surface water level variation. 

GRLM delivers data for every Jason-1 pass that has a 9.9-day period, so the FAS 10-day time step 
requirement is nominally met. Flowever, for various reasons, over many lakes and reservoirs the Jason- 1 
standard product algorithms yield no height information for some passes. For many instances in both 
2004 and 2005, months or even years passed between instances in which water levels were actually 
estimated. These dated sites have archival value, but they no longer provide information about current 
regional water availability. A breakdown of the time to last estimated water level is shown in Figure 8. 
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Figure 8. Histogram of time to last estimated water level checked at the 
end of May 2004 and again at the end of May 2005. 


From May 2004 to May 2005, the number of lakes and reservoirs being monitored increased over 
60 percent from 52 to 84. An incremental increase occurred in the number of water bodies with near-real- 
time monitoring, but most of the increase consisted of lakes and reservoirs that have had little or no valid 
data since Jason- 1 became operational. Much of the difficulty stems from the fact that more near-land 
data is being filtered by new algorithms implemented for standard Jason- 1 data processing. Although 
some indication was noted that this problem was being addressed in the fall of 2004 (NASA, 2004b), little 
or no positive impact seems to have occurred. The geographic distribution of which sites are more current 
and which sites are more out of date is shown in Figure 9. 
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Figure 9. Geographic distribution of time to last estimated water level at the end of May 2005. 
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3.2.3 GRLM Coverage 

In 2004, little coverage existed in South America, western North America, or eastern Asia. Australia had 

no monitored sites. As can be seen in Figure 9, through May 2005 sites have been added in all of these 

areas. Unfortunately, the overwhelming majority of the new sites are badly dated, so the near-real-time 

coverage has changed little. 
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GSFC Goddard Space Flight Center 
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NDVI Normalized Difference Vegetation Index 

OSDR Operational Sensor Data Record 
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